{
 "metadata": {
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  },
  "language_info": {
   "name": "python",
   "version": "3.10.13",
   "mimetype": "text/x-python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "pygments_lexer": "ipython3",
   "nbconvert_exporter": "python",
   "file_extension": ".py"
  },
  "kaggle": {
   "accelerator": "nvidiaTeslaT4",
   "dataSources": [],
   "dockerImageVersionId": 30699,
   "isInternetEnabled": true,
   "language": "python",
   "sourceType": "notebook",
   "isGpuEnabled": true
  }
 },
 "nbformat_minor": 4,
 "nbformat": 4,
 "cells": [
  {
   "cell_type": "code",
   "source": [
    "! pip install langchain openai chromadb tiktoken rank_bm25"
   ],
   "metadata": {
    "execution": {
     "iopub.status.busy": "2024-05-09T21:15:45.284801Z",
     "iopub.execute_input": "2024-05-09T21:15:45.285512Z",
     "iopub.status.idle": "2024-05-09T21:16:01.881836Z",
     "shell.execute_reply.started": "2024-05-09T21:15:45.285476Z",
     "shell.execute_reply": "2024-05-09T21:16:01.880480Z"
    },
    "trusted": true
   },
   "execution_count": 6,
   "outputs": [
    {
     "name": "stdout",
     "text": "Requirement already satisfied: langchain in /opt/conda/lib/python3.10/site-packages (0.1.19)\nRequirement already satisfied: openai in /opt/conda/lib/python3.10/site-packages (1.27.0)\nRequirement already satisfied: chromadb in /opt/conda/lib/python3.10/site-packages (0.5.0)\nRequirement already satisfied: tiktoken in /opt/conda/lib/python3.10/site-packages (0.6.0)\nCollecting rank_bm25\n  Downloading rank_bm25-0.2.2-py3-none-any.whl.metadata (3.2 kB)\nRequirement already satisfied: PyYAML>=5.3 in /opt/conda/lib/python3.10/site-packages (from langchain) (6.0.1)\nRequirement already satisfied: SQLAlchemy<3,>=1.4 in /opt/conda/lib/python3.10/site-packages (from langchain) (2.0.25)\nRequirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /opt/conda/lib/python3.10/site-packages (from langchain) (3.9.1)\nRequirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /opt/conda/lib/python3.10/site-packages (from langchain) (4.0.3)\nRequirement already satisfied: dataclasses-json<0.7,>=0.5.7 in /opt/conda/lib/python3.10/site-packages (from langchain) (0.6.4)\nRequirement already satisfied: langchain-community<0.1,>=0.0.38 in /opt/conda/lib/python3.10/site-packages (from langchain) (0.0.38)\nRequirement already satisfied: langchain-core<0.2.0,>=0.1.52 in /opt/conda/lib/python3.10/site-packages (from langchain) (0.1.52)\nRequirement already satisfied: langchain-text-splitters<0.1,>=0.0.1 in /opt/conda/lib/python3.10/site-packages (from langchain) (0.0.1)\nRequirement already satisfied: langsmith<0.2.0,>=0.1.17 in /opt/conda/lib/python3.10/site-packages (from langchain) (0.1.56)\nRequirement already satisfied: numpy<2,>=1 in /opt/conda/lib/python3.10/site-packages (from langchain) (1.26.4)\nRequirement already satisfied: pydantic<3,>=1 in /opt/conda/lib/python3.10/site-packages (from langchain) (2.5.3)\nRequirement already satisfied: requests<3,>=2 in /opt/conda/lib/python3.10/site-packages (from langchain) (2.31.0)\nRequirement already satisfied: tenacity<9.0.0,>=8.1.0 in /opt/conda/lib/python3.10/site-packages (from langchain) (8.2.3)\nRequirement already satisfied: anyio<5,>=3.5.0 in /opt/conda/lib/python3.10/site-packages (from openai) (4.2.0)\nRequirement already satisfied: distro<2,>=1.7.0 in /opt/conda/lib/python3.10/site-packages (from openai) (1.9.0)\nRequirement already satisfied: httpx<1,>=0.23.0 in /opt/conda/lib/python3.10/site-packages (from openai) (0.27.0)\nRequirement already satisfied: sniffio in /opt/conda/lib/python3.10/site-packages (from openai) (1.3.0)\nRequirement already satisfied: tqdm>4 in /opt/conda/lib/python3.10/site-packages (from openai) (4.66.1)\nRequirement already satisfied: typing-extensions<5,>=4.7 in /opt/conda/lib/python3.10/site-packages (from openai) (4.9.0)\nRequirement already satisfied: build>=1.0.3 in /opt/conda/lib/python3.10/site-packages (from chromadb) (1.2.1)\nRequirement already satisfied: chroma-hnswlib==0.7.3 in /opt/conda/lib/python3.10/site-packages (from chromadb) (0.7.3)\nRequirement already satisfied: fastapi>=0.95.2 in /opt/conda/lib/python3.10/site-packages (from chromadb) (0.108.0)\nRequirement already satisfied: uvicorn>=0.18.3 in /opt/conda/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.25.0)\nRequirement already satisfied: posthog>=2.4.0 in /opt/conda/lib/python3.10/site-packages (from chromadb) (3.5.0)\nRequirement already satisfied: onnxruntime>=1.14.1 in /opt/conda/lib/python3.10/site-packages (from chromadb) (1.17.3)\nRequirement already satisfied: opentelemetry-api>=1.2.0 in /opt/conda/lib/python3.10/site-packages (from chromadb) (1.22.0)\nRequirement already satisfied: opentelemetry-exporter-otlp-proto-grpc>=1.2.0 in /opt/conda/lib/python3.10/site-packages (from chromadb) (1.22.0)\nRequirement already satisfied: opentelemetry-instrumentation-fastapi>=0.41b0 in /opt/conda/lib/python3.10/site-packages (from chromadb) (0.43b0)\nRequirement already satisfied: opentelemetry-sdk>=1.2.0 in /opt/conda/lib/python3.10/site-packages (from chromadb) (1.22.0)\nRequirement already satisfied: tokenizers>=0.13.2 in /opt/conda/lib/python3.10/site-packages (from chromadb) (0.15.2)\nRequirement already satisfied: pypika>=0.48.9 in /opt/conda/lib/python3.10/site-packages (from chromadb) (0.48.9)\nRequirement already satisfied: overrides>=7.3.1 in /opt/conda/lib/python3.10/site-packages (from chromadb) (7.4.0)\nRequirement already satisfied: importlib-resources in /opt/conda/lib/python3.10/site-packages (from chromadb) (6.1.1)\nRequirement already satisfied: grpcio>=1.58.0 in /opt/conda/lib/python3.10/site-packages (from chromadb) (1.60.0)\nRequirement already satisfied: bcrypt>=4.0.1 in /opt/conda/lib/python3.10/site-packages (from chromadb) (4.1.3)\nRequirement already satisfied: typer>=0.9.0 in /opt/conda/lib/python3.10/site-packages (from chromadb) (0.9.0)\nRequirement already satisfied: kubernetes>=28.1.0 in /opt/conda/lib/python3.10/site-packages (from chromadb) (29.0.0)\nRequirement already satisfied: mmh3>=4.0.1 in /opt/conda/lib/python3.10/site-packages (from chromadb) (4.1.0)\nRequirement already satisfied: orjson>=3.9.12 in /opt/conda/lib/python3.10/site-packages (from chromadb) (3.10.3)\nRequirement already satisfied: regex>=2022.1.18 in /opt/conda/lib/python3.10/site-packages (from tiktoken) (2023.12.25)\nRequirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (23.2.0)\nRequirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.0.4)\nRequirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.9.3)\nRequirement already satisfied: frozenlist>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.4.1)\nRequirement already satisfied: aiosignal>=1.1.2 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.1)\nRequirement already satisfied: idna>=2.8 in /opt/conda/lib/python3.10/site-packages (from anyio<5,>=3.5.0->openai) (3.6)\nRequirement already satisfied: exceptiongroup>=1.0.2 in /opt/conda/lib/python3.10/site-packages (from anyio<5,>=3.5.0->openai) (1.2.0)\nRequirement already satisfied: packaging>=19.1 in /opt/conda/lib/python3.10/site-packages (from build>=1.0.3->chromadb) (23.2)\nRequirement already satisfied: pyproject_hooks in /opt/conda/lib/python3.10/site-packages (from build>=1.0.3->chromadb) (1.1.0)\nRequirement already satisfied: tomli>=1.1.0 in /opt/conda/lib/python3.10/site-packages (from build>=1.0.3->chromadb) (2.0.1)\nRequirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /opt/conda/lib/python3.10/site-packages (from dataclasses-json<0.7,>=0.5.7->langchain) (3.21.1)\nRequirement already satisfied: typing-inspect<1,>=0.4.0 in /opt/conda/lib/python3.10/site-packages (from dataclasses-json<0.7,>=0.5.7->langchain) (0.9.0)\nRequirement already satisfied: starlette<0.33.0,>=0.29.0 in /opt/conda/lib/python3.10/site-packages (from fastapi>=0.95.2->chromadb) (0.32.0.post1)\nRequirement already satisfied: certifi in /opt/conda/lib/python3.10/site-packages (from httpx<1,>=0.23.0->openai) (2024.2.2)\nRequirement already satisfied: httpcore==1.* in /opt/conda/lib/python3.10/site-packages (from httpx<1,>=0.23.0->openai) (1.0.5)\nRequirement already satisfied: h11<0.15,>=0.13 in /opt/conda/lib/python3.10/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai) (0.14.0)\nRequirement already satisfied: six>=1.9.0 in /opt/conda/lib/python3.10/site-packages (from kubernetes>=28.1.0->chromadb) (1.16.0)\nRequirement already satisfied: python-dateutil>=2.5.3 in /opt/conda/lib/python3.10/site-packages (from kubernetes>=28.1.0->chromadb) (2.9.0.post0)\nRequirement already satisfied: google-auth>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from kubernetes>=28.1.0->chromadb) (2.26.1)\nRequirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /opt/conda/lib/python3.10/site-packages (from kubernetes>=28.1.0->chromadb) (1.7.0)\nRequirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.10/site-packages (from kubernetes>=28.1.0->chromadb) (1.3.1)\nRequirement already satisfied: oauthlib>=3.2.2 in /opt/conda/lib/python3.10/site-packages (from kubernetes>=28.1.0->chromadb) (3.2.2)\nRequirement already satisfied: urllib3>=1.24.2 in /opt/conda/lib/python3.10/site-packages (from kubernetes>=28.1.0->chromadb) (1.26.18)\nRequirement already satisfied: jsonpatch<2.0,>=1.33 in /opt/conda/lib/python3.10/site-packages (from langchain-core<0.2.0,>=0.1.52->langchain) (1.33)\nRequirement already satisfied: coloredlogs in /opt/conda/lib/python3.10/site-packages (from onnxruntime>=1.14.1->chromadb) (15.0.1)\nRequirement already satisfied: flatbuffers in /opt/conda/lib/python3.10/site-packages (from onnxruntime>=1.14.1->chromadb) (23.5.26)\nRequirement already satisfied: protobuf in /opt/conda/lib/python3.10/site-packages (from onnxruntime>=1.14.1->chromadb) (3.20.3)\nRequirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from onnxruntime>=1.14.1->chromadb) (1.12)\nRequirement already satisfied: deprecated>=1.2.6 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-api>=1.2.0->chromadb) (1.2.14)\nRequirement already satisfied: importlib-metadata<7.0,>=6.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-api>=1.2.0->chromadb) (6.11.0)\nRequirement already satisfied: backoff<3.0.0,>=1.10.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb) (2.2.1)\nRequirement already satisfied: googleapis-common-protos~=1.52 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb) (1.62.0)\nRequirement already satisfied: opentelemetry-exporter-otlp-proto-common==1.22.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb) (1.22.0)\nRequirement already satisfied: opentelemetry-proto==1.22.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb) (1.22.0)\nRequirement already satisfied: opentelemetry-instrumentation-asgi==0.43b0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.43b0)\nRequirement already satisfied: opentelemetry-instrumentation==0.43b0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.43b0)\nRequirement already satisfied: opentelemetry-semantic-conventions==0.43b0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.43b0)\nRequirement already satisfied: opentelemetry-util-http==0.43b0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.43b0)\nRequirement already satisfied: setuptools>=16.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-instrumentation==0.43b0->opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (69.0.3)\nRequirement already satisfied: wrapt<2.0.0,>=1.0.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-instrumentation==0.43b0->opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (1.14.1)\nRequirement already satisfied: asgiref~=3.0 in /opt/conda/lib/python3.10/site-packages (from opentelemetry-instrumentation-asgi==0.43b0->opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (3.8.1)\nRequirement already satisfied: monotonic>=1.5 in /opt/conda/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb) (1.6)\nRequirement already satisfied: annotated-types>=0.4.0 in /opt/conda/lib/python3.10/site-packages (from pydantic<3,>=1->langchain) (0.6.0)\nRequirement already satisfied: pydantic-core==2.14.6 in /opt/conda/lib/python3.10/site-packages (from pydantic<3,>=1->langchain) (2.14.6)\nRequirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests<3,>=2->langchain) (3.3.2)\nRequirement already satisfied: greenlet!=0.4.17 in /opt/conda/lib/python3.10/site-packages (from SQLAlchemy<3,>=1.4->langchain) (3.0.3)\nRequirement already satisfied: huggingface_hub<1.0,>=0.16.4 in /opt/conda/lib/python3.10/site-packages (from tokenizers>=0.13.2->chromadb) (0.22.2)\nRequirement already satisfied: click<9.0.0,>=7.1.1 in /opt/conda/lib/python3.10/site-packages (from typer>=0.9.0->chromadb) (8.1.7)\nRequirement already satisfied: httptools>=0.5.0 in /opt/conda/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.6.1)\nRequirement already satisfied: python-dotenv>=0.13 in /opt/conda/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb) (1.0.0)\nRequirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /opt/conda/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.19.0)\nRequirement already satisfied: watchfiles>=0.13 in /opt/conda/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.21.0)\nRequirement already satisfied: websockets>=10.4 in /opt/conda/lib/python3.10/site-packages (from uvicorn[standard]>=0.18.3->chromadb) (12.0)\nRequirement already satisfied: cachetools<6.0,>=2.0.0 in /opt/conda/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (4.2.4)\nRequirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (0.3.0)\nRequirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.10/site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (4.9)\nRequirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers>=0.13.2->chromadb) (3.13.1)\nRequirement already satisfied: fsspec>=2023.5.0 in /opt/conda/lib/python3.10/site-packages (from huggingface_hub<1.0,>=0.16.4->tokenizers>=0.13.2->chromadb) (2024.2.0)\nRequirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.10/site-packages (from importlib-metadata<7.0,>=6.0->opentelemetry-api>=1.2.0->chromadb) (3.17.0)\nRequirement already satisfied: jsonpointer>=1.9 in /opt/conda/lib/python3.10/site-packages (from jsonpatch<2.0,>=1.33->langchain-core<0.2.0,>=0.1.52->langchain) (2.4)\nRequirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/lib/python3.10/site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain) (1.0.0)\nRequirement already satisfied: humanfriendly>=9.1 in /opt/conda/lib/python3.10/site-packages (from coloredlogs->onnxruntime>=1.14.1->chromadb) (10.0)\nRequirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->onnxruntime>=1.14.1->chromadb) (1.3.0)\nRequirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /opt/conda/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (0.5.1)\nDownloading rank_bm25-0.2.2-py3-none-any.whl (8.6 kB)\nInstalling collected packages: rank_bm25\nSuccessfully installed rank_bm25-0.2.2\n",
     "output_type": "stream"
    }
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain.document_loaders import PyPDFLoader\n",
    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
    "from langchain.vectorstores import Chroma\n",
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "OPENAI_API_KEY = getpass()\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = OPENAI_API_KEY\n",
    "\n",
    "embedding = OpenAIEmbeddings()\n",
    "\n",
    "\n",
    "# Load pdf\n",
    "loader = PyPDFLoader(\"https://arxiv.org/pdf/2309.10305v2.pdf\")\n",
    "data = loader.load()\n",
    "\n",
    "# Split\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)\n",
    "splits = text_splitter.split_documents(data[:6])\n"
   ],
   "metadata": {
    "execution": {
     "iopub.status.busy": "2024-05-09T21:15:20.809406Z",
     "iopub.execute_input": "2024-05-09T21:15:20.809902Z",
     "iopub.status.idle": "2024-05-09T21:15:29.430317Z",
     "shell.execute_reply.started": "2024-05-09T21:15:20.809871Z",
     "shell.execute_reply": "2024-05-09T21:15:29.429156Z"
    },
    "trusted": true
   },
   "execution_count": 3,
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdin",
     "text": " ························································\n"
    }
   ]
  },
  {
   "cell_type": "code",
   "source": [
    "from langchain.retrievers import BM25Retriever, EnsembleRetriever"
   ],
   "metadata": {
    "execution": {
     "iopub.status.busy": "2024-05-09T21:16:01.884078Z",
     "iopub.execute_input": "2024-05-09T21:16:01.884445Z",
     "iopub.status.idle": "2024-05-09T21:16:01.890166Z",
     "shell.execute_reply.started": "2024-05-09T21:16:01.884411Z",
     "shell.execute_reply": "2024-05-09T21:16:01.888964Z"
    },
    "trusted": true
   },
   "execution_count": 7,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "bm25_retriever = BM25Retriever.from_documents(\n",
    "    documents=splits\n",
    ")\n",
    "bm25_retriever.k = 4"
   ],
   "metadata": {
    "execution": {
     "iopub.status.busy": "2024-05-09T21:16:01.891576Z",
     "iopub.execute_input": "2024-05-09T21:16:01.891917Z",
     "iopub.status.idle": "2024-05-09T21:16:01.916205Z",
     "shell.execute_reply.started": "2024-05-09T21:16:01.891888Z",
     "shell.execute_reply": "2024-05-09T21:16:01.915143Z"
    },
    "trusted": true
   },
   "execution_count": 8,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "vectordb = Chroma.from_documents(documents=splits, embedding=embedding)\n",
    "\n",
    "retriever = vectordb.as_retriever(search_kwargs={\"k\": 4})"
   ],
   "metadata": {
    "execution": {
     "iopub.status.busy": "2024-05-09T21:16:12.506570Z",
     "iopub.execute_input": "2024-05-09T21:16:12.507517Z",
     "iopub.status.idle": "2024-05-09T21:16:17.183046Z",
     "shell.execute_reply.started": "2024-05-09T21:16:12.507482Z",
     "shell.execute_reply": "2024-05-09T21:16:17.182166Z"
    },
    "trusted": true
   },
   "execution_count": 9,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "ensemble_retriever = EnsembleRetriever(\n",
    "    retrievers=[bm25_retriever, retriever], weights=[0.5, 0.5]\n",
    ")"
   ],
   "metadata": {
    "execution": {
     "iopub.status.busy": "2024-05-09T21:16:25.599789Z",
     "iopub.execute_input": "2024-05-09T21:16:25.600182Z",
     "iopub.status.idle": "2024-05-09T21:16:25.605456Z",
     "shell.execute_reply.started": "2024-05-09T21:16:25.600153Z",
     "shell.execute_reply": "2024-05-09T21:16:25.604377Z"
    },
    "trusted": true
   },
   "execution_count": 10,
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "docs = ensemble_retriever.invoke(\"What is baichuan2 ？\")\n",
    "docs"
   ],
   "metadata": {
    "execution": {
     "iopub.status.busy": "2024-05-09T21:16:53.329317Z",
     "iopub.execute_input": "2024-05-09T21:16:53.329709Z",
     "iopub.status.idle": "2024-05-09T21:16:53.525150Z",
     "shell.execute_reply.started": "2024-05-09T21:16:53.329679Z",
     "shell.execute_reply": "2024-05-09T21:16:53.523966Z"
    },
    "trusted": true
   },
   "execution_count": 12,
   "outputs": [
    {
     "execution_count": 12,
     "output_type": "execute_result",
     "data": {
      "text/plain": "[Document(page_content='users and tasks, the specific behavior of each\\ntask is unpredictable, often leading to idle GPU\\nnodes within the cluster. Considering that a single\\nmachine equipped with eight A800 GPUs could\\nadequately meet the memory requirements for our\\nBaichuan 2-7B and Baichuan 2-13B models, the\\n4https://scipy.org/primary design criterion for our training framework\\nis the machine-level elasticity, which supports that\\nresources for tasks can be dynamically modified', metadata={'source': 'https://arxiv.org/pdf/2309.10305v2.pdf', 'page': 5}),\n Document(page_content='In this technical report, we introduce Baichuan\\n2, a series of large-scale multilingual language\\nmodels. Baichuan 2 has two separate models,\\nBaichuan 2-7B with 7 billion parameters and\\nBaichuan 2-13B with 13 billion parameters. Both\\nmodels were trained on 2.6 trillion tokens, which\\nto our knowledge is the largest to date, more than\\ndouble that of Baichuan 1 (Baichuan, 2023b,a).\\nWith such a massive amount of training data,\\nBaichuan 2 achieves significant improvements over', metadata={'page': 1, 'source': 'https://arxiv.org/pdf/2309.10305v2.pdf'}),\n Document(page_content='embedding becomes smaller during training which\\ndisturb the training dynamics. NormHead can\\nstabilize the dynamics significantly. Second, we\\nfound that the semantic information is mainly\\nencoded by the cosine similarity of Embedding\\nrather than L2 distance. Since the current linear\\nclassifier computes logits by dot product, which\\nis a mixture of L2 distance and cosine similarity.\\nNormHead alleviates the distraction of L2 distance\\nin computing logits. For more details, please refer\\nappendix B.', metadata={'source': 'https://arxiv.org/pdf/2309.10305v2.pdf', 'page': 4}),\n Document(page_content='demonstrates strong performance on medical and\\nlegal domain tasks. On benchmarks such as\\nMedQA (Jin et al., 2021) and JEC-QA (Zhong\\net al., 2020), Baichuan 2 outperforms other open-\\nsource models, making it a suitable foundation\\nmodel for domain-specific optimization.\\nAdditionally, we also released two chat\\nmodels, Baichuan 2-7B-Chat and Baichuan 2-\\n13B-Chat, optimized to follow human instructions.\\nThese models excel at dialogue and context\\nunderstanding. We will elaborate on our', metadata={'page': 1, 'source': 'https://arxiv.org/pdf/2309.10305v2.pdf'}),\n Document(page_content='However, most open-source large language\\nmodels have focused primarily on English. For\\ninstance, the main data source for LLaMA\\nis Common Crawl1, which comprises 67% of\\nLLaMA’s pre-training data but is filtered to English\\ncontent only. Other open source LLMs such as\\nMPT (MosaicML, 2023) and Falcon (Penedo et al.,\\n2023) are also focused on English and have limited\\ncapabilities in other languages. This hinders the\\ndevelopment and application of LLMs in specific\\nlanguages, such as Chinese.', metadata={'source': 'https://arxiv.org/pdf/2309.10305v2.pdf', 'page': 1}),\n Document(page_content='Baichuan 1. On general benchmarks like MMLU\\n(Hendrycks et al., 2021a), CMMLU (Li et al.,\\n2023), and C-Eval (Huang et al., 2023), Baichuan\\n2-7B achieves nearly 30% higher performance\\ncompared to Baichuan 1-7B. Specifically, Baichuan\\n2 is optimized to improve performance on math\\nand code problems. On the GSM8K (Cobbe\\net al., 2021) and HumanEval (Chen et al., 2021)\\nevaluations, Baichuan 2 nearly doubles the results\\nof the Baichuan 1. In addition, Baichuan 2 also', metadata={'page': 1, 'source': 'https://arxiv.org/pdf/2309.10305v2.pdf'}),\n Document(page_content='To fit the scaling law of the model, we employed\\nthe formula given by Henighan et al. (2020):\\nLC=a×Cb+L∞ (2)\\nwhere L∞is the irreducible loss and the first\\nterm is the reducible loss which is formulated as a\\npower-law scaling term. Care training flops and\\ntheLCare final loss of the model in that flops. We\\nused the curve_fit function from the SciPy4\\nlibrary to fit the parameters. The final fitted scaling\\ncurve and the predicted 7 billion and 13 billion', metadata={'source': 'https://arxiv.org/pdf/2309.10305v2.pdf', 'page': 5}),\n Document(page_content='Baichuan 1-13B-Base 52.40 51.60 55.30 49.69 43.20 43.01 26.76 11.5913B\\nBaichuan 2-13B-Base 58.10 59.17 61.97 54.33 48.17 48.78 52.77 17.07\\nTable 1: Overall results of Baichuan 2 compared with other similarly sized LLMs on general benchmarks. * denotes\\nresults derived from official websites.\\nFigure 1: The distribution of different categories of\\nBaichuan 2 training data.\\nData processing : For data processing, we focus\\non data frequency and quality. Data frequency', metadata={'page': 2, 'source': 'https://arxiv.org/pdf/2309.10305v2.pdf'})]"
     },
     "metadata": {}
    }
   ]
  },
  {
   "cell_type": "code",
   "source": [],
   "metadata": {},
   "execution_count": null,
   "outputs": []
  }
 ]
}
